We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building in well-defined settings, and evaluate only a limited set of research artifacts (e.g., CSV submission files). To make AI agent benchmarking more relevant to the practice of machine learning engineering, our framework scales along two critical dimensions. First, recognizing that effective ML engineering requires a range of diverse skills, TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and tasks. We design challenges to evaluate both isolated capabilities (including data handling, understanding research repositories, and code translation) and their combinations, and rather than addressing each challenge independently, we develop tools that support designing multiple challenges at scale. Second, we implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models, using both precise numeric measures and more flexible LLM-based evaluation approaches. This dual strategy balances objective assessment with contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. We open-source our benchmarking framework to facilitate future research on the ML engineering capabilities of AI agents.

Machine learning engineering is the field of developing, implementing, and maintaining machine learning systems. It involves the application of engineering principles to the design, development, and deployment of machine learning models, algorithms, and applications. The primary focus of ML engineering is to build scalable and efficient machine learning systems that can process large volumes of data and generate accurate predictions. It involves various tasks such as data preparation, model development, model training, model deployment, and model monitoring. ML engineering requires a combination of skills in computer science, mathematics, statistics, and domain-specific knowledge.

In spite of machine learning's rapid growth, its engineering support is scattered in many forms, and tends to favor certain engineering stages, stakeholders, and evaluation preferences. We envision a capability-based framework, which uses fine-grained specifications for ML model behaviors to unite existing efforts towards better ML engineering. We use concrete scenarios (model design, debugging, and maintenance) to articulate capabilities' broad applications across various different dimensions, and their impact on building safer, more generalizable and more trustworthy models that reflect human needs. Through preliminary experiments, we show capabilities' potential for reflecting model generalizability, which can provide guidance for ML engineering process. We discuss challenges and opportunities for capabilities' integration into ML engineering.

Our pipeline can be broken down into two sub-pipelines, the data-processing pipeline which prepares the data for the model, and the data-science pipeline which implements the model and its functions. The SVD algorithm from Scikit-Surprise is trained on about 1.2 Million Ratings for 50k users and 20k items.

For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem. Ad clicks constitute a significant class of user engagements and are often used as the primary signal for the usefulness of ads to users. Additionally, in cost-per-click advertising systems where advertisers are charged per click, click rate expectations feed directly into value estimation. Accordingly, CTR model development is a significant investment for most Internet advertising companies. Engineering for such problems requires many machine learning (ML) techniques suited to online learning that go well beyond traditional accuracy improvements, especially concerning efficiency, reproducibility, calibration, credit attribution. We present a case study of practical techniques deployed in Google's search ads CTR model. This paper provides an industry case study highlighting important areas of current ML research and illustrating how impactful new ML methods are evaluated and made useful in a large-scale industrial setting.

ML engineering and data science are not the same things, and here's why: you know when people say that data science is a mix between business knowledge, statistics, and computer science? Well, ML engineering is a lot more about computer science and less about statistics and business knowledge. In practice, that means that a data scientist is more talented at creating new models, analysing and interpreting data, and understanding the mathematical basis for those models. He or she will usually come from a statistics background, some might have a PhD, and will be really good at math, while programming was something they learned in order to do math using a computer. An ML engineer, on the other hand, will shine at building and optimising data flows, implementing models, and putting them in production.

Artificial intelligence was almost exclusively the domain of academic research for decades. In the past ten years, however, machine learning (ML) techniques have finally achieved sufficient effectiveness and practicality for large-scale adoption in companies and institutions. This adoption, however, remains incipient. Most organizations are still in the early stages gaining proficiency in these technologies and growing them at enterprise scale. The potential for professionals in this area, therefore, is enormous, evinced by the steady increase in ML job openings and courses. Given the proliferation of ML jobs postings out there, what are the roles and positions in the field?

Have you mastered the art of building and training ML models, and are now ready to use them in a production deployment for a product or service? If so, we have a new set of courses to get you going. Built as a collaboration between the TensorFlow team, Andrew Ng, and deeplearning.ai, The new specialization builds on the foundational knowledge taught in the popular specialization, DeepLearning.AI TensorFlow Developer Professional Certificate, that teaches how to build machine learning models with TensorFlow. The new MLOps specialization kicks off with an introductory course taught by Andrew Ng, followed by courses taught by Robert Crowe and Laurence Moroney that dive into the details of getting your models out to users.

Software Engineering, as a discipline, has matured over the past 5 decades. The modern world heavily depends on it, so the increased maturity of Software Engineering was an eventuality. Practices like testing and reliable technologies help make Software Engineering reliable enough to build industries upon. Meanwhile, Machine Learning (ML) has also grown over the past 2 decades. ML is used more and more for research, experimentation and production workloads. But ML Engineering, as a discipline, has not widely matured as much as its Software Engineering ancestor. Can we take what we have learned and help the nascent field of applied ML evolve into ML Engineering the way Programming evolved into Software Engineering? In this article we will give a whirlwind tour of Sibyl and TensorFlow Extended (TFX), two successive end-to-end (E2E) ML platforms at Alphabet. We will share the lessons learned from over a decade of applied ML built on these platforms, explain both their similarities and their differences, and expand on the shifts (both mental and technical) that helped us on our journey.

The field of Machine Learning (ML) has been consistently evolving since Data Science started gaining traction in 2012. However, I believe 2018 was a critical inflection point in the ML industry. After helping Insight Fellows build dozens of ML products to get roles on applied ML teams, and reading through both corporate and academic published research and, I've seen more need for engineering skills than ever before. As a field that has consistently toed the line between its origins in academic research and the need to serve customer needs, it has often been hard to reconcile engineering standards with ML models. As both research and applied teams are doubling down on their engineering and infrastructure needs, the nascent field of ML Engineering will build upon 2018's foundation and truly blossom in 2019.